Fast and reliable hand pose estimation has a wide application in the fields such as human-computer interaction. In order to deal with the influences to the hand pose estimation caused by the light intensity changes, self-occlusions and large pose variations, a deep network framework based on label distribution learning was proposed. In the network, the point cloud of the hand was used as the input data, which was normalized through the farthest point sampling and Oriented Bounding Box (OBB). Then, the PointNet++ was utilized to extract features from the hand point cloud data. To deal with the highly non-linear relationship between the point cloud and the hand joint points, the positions of the hand joint points were predicted by the label distribution learning network. Compared with the traditional depth map based approaches, the proposed method was able to effectively extract discriminative hand geometric features with low computation cost and high accuracy. A set of tests were conducted on the public MSRA dataset to verify the effectiveness of the proposed hand pose estimation network. Experimental results showed that the average error of the hand joints estimated by this network was 8.43 mm, the average processing time of a frame was 12.8 ms, and the error of pose estimation was reduced by 11.82% and 0.83% respectively compared with the 3D CNN and Hand PointNet.